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(57) Abstract: A method and system for time-motion compensation for use with head mounted displays is disclosed. Acconiing to 
the method, a remote camera c^)tiires an image for display on a head-mounted display (HMD) including a viewing window. The 
image and camera position data is transmitted to a system including the HMD for display to a wearer of the HMD. The HMD position 
is determined. An offset between the HMD position and a known position of the HMD is determined as is an offset between the 
camera position and a known position of the camera. The image is offset relative to the viewing window of the image based on the 
difference between the two determined offsets. 
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Method and System for Time/motion Compensation for Head Mounted Displays 



Field of the Invention 

The present invention generally relates to telepresence systems and more 
particularly relates to motion compensation in telepresence systems. 

5 Background of the Invention 

The field of remote control has come a long way since the days of watching a 
model aircraft fly under the control of a handheld controller. Robotics and remote robotic 
manipulation have created a strong and pressing need for more remote and better remote 
control systems. Obviously, an ideal form of remote control involves providing an 
10 operator with all the sensations of operating the remote robot without the inherent 
dangers, travel, and so forth. In order to achieve this, a telepresence system is used. 

Telepresence systems are sensory feedback systems for allowing sensing and 
monitoring of remote systems. A typical telepresence sensor is a camera and a head 
mounted display. The system provides visual feedback from a remote location to an 

15 operator. For example, in a telepresence system for an automobile, the front windshield is 
provided with a camera. The controls of the vehicle are provided with actuators for 
automatically manipulating same. An operator is provided with a duplicate of the cabin of 
the car. The windshield is replaced with a display and the controls are linked via 
communications to the actuators within the vehicle. Turning of the steering wheel in the 

20 cabin of the car causes the steering wheel to turn in the vehicle. Similarly, the camera 
captures images in front of the car and they are displayed on the display in the cabin of 
the car. 

Presently, there is a trend toward providing the visual feedback using a head 
mounted display (HMD). A head mounted display is a small display or two small displays 

25 mounted for being worn on a users head. Advantageously, an HMD with two displays 
provides stereo imaging allowing a user to perceive depth of field. Alternatively, such an 
HMD provides two identical images, one to each display. Unfortunately, the head 
mounted display only presents a user with information from approximately in front of the 
user. Thus, when a user turns their head, the image seen and the expected image differ. 

30 Therefore, the camera is mounted on a mechanism which moves in accordance widi 
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detected HMD movement. Thus, the image before the user is in accordance with the 
user's head position. 



Generally, it is an object of telepresence systems to provide a visual sensation of 
being in the place of the robot and a control system for controlling the robot as well. 
5 Thus, telepresence systems aim to provide feedback that is appropriate to different 
situations. 

Unfortunately, a camera does not move in exact synchronisation with the HMD so 
the image is not perfectly aligned with the expectations of the user during head motion. 
This misalignment can result in disorientation and nausea on the part of an operator. 

10 The disclosures in U.S. Patent 5,579,026 issued on Nov. 26, 1996 in the name of 

Tabata and in U.S. Patent 5,917,460 issued on Jun. 29, 1999 in the name of Kodama 
focus on image display for use in, for example, virtual reality and games. In there is 
described a head mounted display in which the position of the projected image can be 
displaced in response to a control unit or in response to the rotational motion of the 

1 5 operator's head. The essence of the head-tracking implementation is that from the user*s 
perspective, the image can be made to remain substantially stationary in space during 
head movements, by being manipulated in a manner opposite to the movements. 
Significantly, the patents do not relate to visual telepresence using slaved cameras. In the 
slaved camera implementation, the camera should follow the motion of the HMD and, as 

20 such, compensation for HMD motion is unnecessary since the image is always of a 
direction in which the head is directed. 

Further because U.S. Patent 5,579,026 relates to displaying a simulated planar 
image, such as a simulation of a television screen located in virtual space in front of the 
user, the patent provides for a fixed frame of reference relative to a wearer of the HMD. 
25 The images in any direction are simulated thus being formed as needed. Unfortunately, in 
telepresence systems, often the video data relating to a particular direction of view is 
unavailable. This complicates the system significantly and as such, the prior art relating to 
video data display is not truly applicable and, one of skill in the art would not refer to 
such. 

30 In U.S. Patent 5,917,460 issued on Jun. 29, 1999 in the name of Kodama a system 

addressing the three-axes displacement (up/dovm, left/right, frontwards/backwards) of a 
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HMD is provided. The displacement appears to linear and is accommodated through a 
mechanical mechanism. The displays are moved in response to detected movement of a 
head and as such, objects remain somewhat stationary from the visual perspective of the 
user. 

5 It is not well suited to use in telepresence wherein a camera tracks the motion of 

the HMD. One of skill in the art, absent hindsight, would not be drawn to maintaining a 
visual reference when a head is turned, for a telepresence system wherein a camera is 
rotated in response to head movement. Of course, the different problem results in a 
different solution. 

1 0 For example, in telepresence systems, the delay between camera image capture 

and head motion is often indeterminate. It is not a workable solution to implement the 
system of the above referenced patents to solve this problem. Because of the unknown 
delays caused by camera response time and communication delays, the solution is not 
trivial. 

1 5 In U.S. Patent 5,933,1 25 a system is disclosed using prediction of the head 

movement to pre-compensate for the delay expected in the generation of a virtual image, 
nominally in a simulated environment. By this means, a time lag in the generation of 
imagery is compensated for by shifting the scene to provide a stable visual frame of 
reference. This method is applicable to short delays and small displacements, where head 

20 tracking information can be used to predict the next head position with reasonable 

accuracy. The patent discloses 100msec as a normal value. Effective prediction of head 
motion is aided by comprehensive information about head movement, including angular 
head velocity and angular acceleration. For small head movements, errors induced are 
small. Typically, these occur in a small period of time. The disclosed embodiments rely 

25 on knowledge of the time delay, which is nominally considered to be constant. 

Unfortunately, when the time delays grow large allowing for substantial motion of a head, 
the errors in the predictive algorithm are unknown and the system is somewhat 
unworkable. 

Furthermore, US 5,933,125 cannot compensate for unanticipated image 
30 movement, only that which occurs m correct response to the operator's head movement. 
Also, it does not relate to visual telepresence systems using remote slave cameras. 
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It would be highly advantageous to provide a system that does not rely on any 
form of prediction for compensation and which works with variable delays between 
image capture and image display. 

S Object of the Invention 

In order to overcome these and other shortcomings of the prior art, it is an object 
of the invention to provide a method of compensating for time delays between head 
motion and camera motion in telepresence systems. 

Summary of the Invention 

10 The invention relates to a method and apparatus that provides a wearer of an 

HMD with a stable frame of visual reference in cases where there may be time delays or 
unwanted motion within the visual capture/visual display systems. 

According to the invention, in order to eliminate some of the disorientation caused 
by time delays in camera motion when a head motion occurs, an image shown on the 

15 display of a head-mounted display (HMD) is offset relative to the field of view of the 
HMD until the camera position is again synchronised with the HMD position. Offsetting 
of the image results in areas of the display for which no image information is available. 
These display areas are provided fill data in the form of a solid shading or some feature 
set for providing visual cues. When the transformed images again overlap the display, the 

20 fill is no longer necessary. 

In accordance with the invention there is provided a method of motion 
compensation for head mounted displays. The method includes the following steps: 
providing an image from an image capture device to a head mounted display including a 
monitor having a field of view; providing camera position data associated with the image; 
25 providing head position data; adjusting the image location relative to the field of view of 
the monitor in accordance with the camera position data and the head position data; and, 
displaying portions of the image at the adjusted locations, those portions remaining within 
the field of view. 

Typically position data includes at least one of orientation data and location data. 
30 Location data is also referred to as displacement data. Typically, portions of the field of 
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view without image data are filled with a predetermined fill. When none of the image data 
is for display within the field of view, the entire field of view is filled with the 
predetermined fill. 

For example, the image is adjusted by the following steps: determining an offset 
5 between the head mounted display position and the camera position; and, offsetting the 
image such that it is offset an amount equal to the offset between the head mounted 
display position and the camera position. 

Advantageously, such a system is not limited by the accuracy of a predictive 
process nor by the time delay between image capture and image display. Instead, it is 
10 reactive, and uses sensed information on HMD position and camera position to formulate 
a transformation for the captured image. The present invention has no limit to the time 
delays for which compensation is possible since the required head position infonmation 
and camera position information are sensed at different times allowing compensation for 
any delay between sensing one and then sensing the other. 

15 Further advantageously, the present invention requires no knowledge of the time 

delay in the system and functions properly in the presence of non-constant time delays. 
There is no requirement that the time delay be measured and it is not used in determining 
the transform of the image. 

Brief Description of the Drawings 

20 The invention will now be described in conjunction with the drawings in which: 

Fig. 1 is a simplified block diagram of a system incorporating an HMD coupled to a first 
computer and in communication with a remote camera; 

Fig. 2 is a simplified diagram showing axes of movement of the systems involved; 

Fig. 3a is a sunplified diagram showing a simulated view of an image appearing within 
25 the field of view of an HMD; 

Fig. 3 b is a simplified diagram showing a simulated view of a portion of the image of Fig. 
3a offset vertically within the field of view of an HMD in response to a downward motion 
of a user's head; 

5 
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Fig. 3c is a simplified diagram showing a simulated view of a portion of the image of Fig. 
3a offset horizontally within the field of view of an HMD in response to a lateral motion 
of a user's head; 

Fig. 3d is a simplified diagram showing a simulated view of a portion of the image of Fig. 
5 3a tilted within the field of view of an HMD in response to a tilting motion of a user's 
head; 

Fig. 3e is a simplified diagram showing a simulated view of a portion of the image of Fig. 
3a tilted and offset both vertically and horizontally within the field of view of an HMD in 
response to a tilting motion combined with a lateral and a horizontal motion of a user's 
10 head; 

Fig. 4a is a simplified diagram showing a simulated view of an image appearing within 
the field of view of an HMD; 

Fig. 4b is a simplified diagram showing a simulated view of a portion of the image of Fig. 
4a offset horizontally within the field of view of an HMD in response to a lateral motion 
15 of a user's head; 

Fig. 4c is a simplified diagram showing a simulated view of an image appearing within 
the field of view of an HMD including a portion of the image of Fig. 4a as well as 
additional image data captured and displayed within the field of view of an HMD when 
the camera motion is partially caught up with the lateral motion of a user's head; 

20 Fig. 4d is a simplified diagram showing a simulated view of an image appearing within 
the field of view of an HMD including a portion of the image of Fig. 4a as well as 
additional image data captured and displayed within the field of view of an HMD after 
the camera motion is fully caught up with the lateral motion of a user's head; 

Fig. 5 is a simplified block diagram of a system incorporating an HMD coupled to a first 
25 computer and in communication across a network with a second computer coupled to a 
remote camera; 

Fig. 6 is a simplified flow diagram of a method according to the invention; 
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Fig. 7 is a simplified block diagram of a telepresence system communicating via a 
satellite communications link; 

Fig. 8 is an image captured by a remote camera as captured; 

Fig. 9 is an image of the same image as that of fig. 8 transformed within an endless image 
5 display space; and, 

Fig. 10 is an image of the same image as that of fig. 9 as displayed on a display having a 
finite image display space. 

Detailed Description of the Invention 

The present invention is described with reference to telepresence systems 
10 operating over long distances such that significant time delays occur between head 
motion and image display of an image for a current head orientation. It is, however, 
equally applicable when a camera drive mechanism provides insufficient response rate to 
allow comfortable viewing of images during normal head motion. It is also applicable in 
situations where unwanted and unmodeled motion of the camera is possible, such as when 
1 5 the camera is mounted on a moving platform. 

Referring to Fig. I, a simplified block diagram of a telepresence system is shown. 
A head mounted display (HMD) 1 including a three-axes head tracker 3 is worn by an 
operator 5. The HMD 1 is coupled with a first computer 7 and provides to the first 
computer 7 HMD values for the HMD position in the form of pitch 21, yaw 22, and roll 

20 23 angles of the HMD 1 as shown in Fig. 2. Of course, since the HMD I is being worn by 
an operator 5, these HMD values relate to the head position of the operator 5. These 
values are provided to the first computer 7 at intervals, preferably more than 100 times 
each second, though other intervals are also acceptable. The HMD values are converted 
by the first computer 7 into control values for controlling positioning of a camera 1 1 . The 

25 control values are transmitted to a mechanism 13 for pointing the camera in order to 
affect camera orientation. As is seen in Figure 2, the mechanism controls pitch 24, yaw 25 
and roll 26 of the camera 11. In response to the control values, the camera position moves 
in accordance with the movement of the HMD 1 . 
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When the mechanism 13 for pointing the camera is physically coupled to the first 
computer 7, the camera 1 1 begins to move when HMD motion is detected. The lag 
between camera motion and HMD motion is determined by communication delays, which 
are very small, processing delays, which may be minimal, and pointing mechanism 
5 performance, which varies. These delays often result in an image provided from the 
camera 1 1 remaining static while the HMD 1 is in motion or moving substantially slower 
than the HMD motion. Of course, since the operator's mind expects motion within the 
visual frame, this is disconcerting and often results in nausea and disorientation. 

This problem is even more notable when communication delay times are 
10 significant such as when used for terrestrial control of systems in space. There, the delay 
is in the order of seconds and, as such, the disorientation of an operator during HMD 
motion is significant. Significantly, disorientation is a cause of operator fatigue resulting 
in limited operator use of a system or limited use of a system during a day. 

Referring again to Fig. 1, the camera 1 1 is constantly acquiring images at a video 
1 5 data capture rate. Each image is transmitted to the first computer for processing, if 
required, and for provision to the HMD 1. According to the invention, the remote system 
also provides camera position information to the first computer 7 and associated with 
each image. Thus, each frame received by the first computer 7 has associated therewith 
camera position information. The camera position information is preferably relative to a 
20 known orientation. Alternatively, it is transformed by the first computer 7 into position 
information relative to a known camera orientation and in a coordinate space analogous to 
that of the HMD 1. 

The HMD position values are used to determine a current HMD orientation in a 
coordinate space analogous to that of the camera 1 1. As such, an offset between camera 

25 orientation and HMD orientation is determinable. Since the HMD 1 is being worn by an 
operator 5 the HMD orientation is directly correlated to the position of the head of the 
operator 5. Of course, the direct correlation is related to sensed position data and in use is 
generally an approximate direct correlation due to a refresh rate of the HMD position 
sensor. The offset between the camera orientation and the HMD orientation is related to a 

30 delay between the local system and the remote system. 
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Therefore when a non-zero offset is determined, the first computer offsets the 
image provided by the camera relative to the field of view of the HMD in order to 
compensate for the determined offset. Referring to Figs. 3a, 3b, 3c, 3d and 3e, some 
examples of image displays are shown. In Fig. 3a, the image is shown for a zero offset 
5 between camera orientation and HMD orientation. This is the steady state of the feedback 
system since the HMD 1 is directed in a same direction as the camera 1 1 and the image 
displayed on the display within the HMD 1 is the same as the image captured by the 
camera. When the HMD orientation is angled down from the camera position, the image 
is offset in a vertical direction as shown in Fig. 3 b. When the camera orientation is offset 
10 horizontally, the image is offset horizontally as shown in Fig, 3c. In Figs. 3d and Fig. 3e, 
the image is rotated and offset relative to the field of view because the camera orientation 
is rotated and offset relative to the HMD orientation. 

Though the described instantaneous corrections shown in Figs. 4a, 4b, 4c and 4d 
appear simple, the steady state nature of the system requires an ever changing imaging 
15 perspective and display perspective. Thus a comparison is necessary between two 
dynamic sets of sensed position data. 

Referring to Figs. 4a, 4b, 4c and 4d, field of view is shown for the HMD 1 during 
a left turn of the operator's head. At first (before the head turn) in a steady state, the exact 
image captured by the camera 1 1 is shown in the display of the HMD 1 at Fig. 4a. When 

20 the head turns, the operator 5 "expects" the image to move to the right since the image is 
not part of the operator 5 and is within their field of view. This expectation is either 
conscious or unconscious. Imagming that the image remains static as the HMD moves, it 
is clear that disorientation would result since individuals take cues fi-om their visual field 
of view during head movement. In order to provide the operator 5 with the "expected" 

25 displacement of objects in die image, the image is offset to a location approximately the 
same as the orientation difference between the HMD 1 and the camera 1 1. For example, 
in the image of Fig. 4b the lighthouse is shifted out of the field of view by the rotation of 
the head. Turning the HMD a degrees, an operator expects static objects within the field 
of view, such as a lighthouse, to shift a degrees within the field of view. This is important 

30 to maintaining comfort of the operator in their personal vision system (their eyes and their 
mind). At a same time, the camera begins to move to match its orientation to that of the 
HMD. Thus, as shown in the image of Fig. 4c more of the scene within the operator's 

9 
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field of view is now available from the camera 11 . As the camera orientation "catches up" 
with the HMD orientation, the field of view of the camera and that of the HMD overlap 
more. When the camera 1 1 is "caught up," the field of view of the HMD again shows an 
entire image captured by the camera as shown in the image of Fig. 4d. 

5 Referring to Fig, 5, another embodiment of the invention is shown for use on a 

network. Here for example, two computers 7 and 15 communicate via a network or 
networks 17. The first computer 7 includes the HMD 1 as a peripheral thereof. The 
second computer 15 includes the camera 1 1 and mechanism 13 for pointing the camera as 
peripherals thereof. Here the processing is performed in either of the first computer 7 or 
10 the second computer 15 though the image processing is preferably performed in the first 
computer 7 in case of networic delays that could cause image offset and result in 
disorientation of an operator 5. Of course, when network delays are known to be 
significant, it is important that image processing is performed on the first computer. 

Referring to Fig. 6, a simplified flow diagram of a method of performing the 
15 invention is shown. An image is captured by the camera 1 1. A sensor captures position 
data in the form of camera orientation values for pitch, roll and yaw. The position data is 
preferably captured concurrently with the image. Alternatively, it is captured 
approximately at a same time but offset by a finite amount either immediately after image 
capture or immediately before. The position data is then associated with the image data. A 
20 simple method of associating the data is by encoding the position data with the image 
data either as header or trailer information. Of course, the image and the position data 
could also be identified with an associating identifier such as an image frame number. 
Alternatively, the two data are transmitted in parallel in a synchronous environment. 

The image and position data are then transmitted to the first computer 7. When the 
25 image and position data are received, they are prepared for processing at the first 
computer 7. Then, the position data of the HMD 1 is acquired by the first computer 7 and 
is used to transform the image in accordance with the invention. The transformed image 
is provided to the display and is displayed thereon to the operator 5. Because the HMD 
position data is gathered immediately before it is needed, the delay between HMD 
30 position data capture and display of the transformed image is very small and results in 
little or no operator disorientation. 

10 
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Concurrently, position data is provided to the mechanism 13 at intervals and the 
mechanism moves the camera 1 1 in accordance with received position data and a current 
orientation of the camera 1 1 . 

Typically, the step of transformmg the image comprises the following steps, some 
5 of which are performed in advance. A correlation between angular movement and display 
or image pixels is determined such that an offset of a degrees results in displacement of 
the image by N pixels in a first direction and by M pixels in a second other direction. A 
transform for rotating the image based on rotations is also determined. Preferably, the 
transforms are sufficiently simple to provide fast image processing. That said, a small 
10 image process;ing delay, because it forms substantially the delay in displaying the data, is 
acceptable. 

Once the image data is received, it is stored in memory for fast processing thereof. 
The HMD position data is acquired and is compared to the camera position data. The 
difference is used in performing the transform to correct the image position for any HMD 
15 motion unaccounted for by the mechanism 13, as of yet. Also, the method corrects for 
unintentional movements of the camera 11 when the camera position sensor is 
independent of the mechanism 13, for example with an inertial position sensor. 

In the above embodiment, a general purpose processor is used to transform the 
image. In an altemative embodiment, a hardware implementation of the transform is used. 
20 A hardware implementation is less easily modified, but has a tremendous impact on 
performance. Using parallel hardware transformation processors, an image can be 
transformed in a small fraction of the time necessary for performing a software 
transformation of the image. 

Referring to Fig. 7 a satellite based telepresence system is shown. Here delays in 
25 the order of seconds between head motion and image display result. Further, the delays 
are not always predictable. Here an HMD 101 is shown positioned on the head of an 
operator 5. The HMD is provided with a head tracker 103 for sensing position data 
relative to the HMD. The HMD is also coupled with a computer 107 for providing display 
data to the HMD and for providing the HMD position data to a communications link 108. 
30 The communications link 108 uplinks the HMD position data to a satellite 117 from 
which it is transmitted to a transceiver 1 16. A computer 1 15 in communication with the 

11 
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transceiver provides the data to a gimbal 113 for repositioning a camera III in 
accordance therewith. The camera 111 captures images which are provided along a 
reverse conmiunication path - computer 115, transceiver 116, satellite 117, 
communications link 108 - to the computer 107. Optionally, a different return path is 
5 used. There the image data is processed for display within the HMD 101. With the 
images, camera position data sensed by a sensor 114 is also provided. The camera 
position data is associated with an image or images captured at approximately a time the 
camera 1 1 1 was in that sensed position. 

The computer 107 uses the camera position data and the image along with data 
10 received from the head tracker 103 to transform the image in accordance with the 
invention. As is evident, the delay between HMD motion and camera motion is 
measurable in seconds. The delay between camera image capture and receipt of the image 
at the computer 107 is also measurable in seconds. As such, significant disorientation of 
the user results absent application of the present invention. 

15 Referring to Fig. 8, an image captured by the camera 1 1 1 is shown. The image is 

displayed as captured when the HMD and the camera orientations are aligned, the camera 
orientation at a time of hnage capture and the HMD orientation at a time of display. If the 
orientations are offset one from another, the image is shifted within the field of view of 
the operator as shown in Fig. 9. Since there is no image data beyond the camera imaging 

20 field of view, the remainder of the display area is shaded with a neutral colour such as 
gray. Altematively, the remainder of the display area is shaded to provide reference 
points to further limit disorientation of the operator 105. Further altematively, the portion 
of the field of view for which no image data is available is left blank. Typically blank 
areas are black in order not to distract an operator. Referring to Fig. 10, when the camera 

25 111 orientation is "caught up'* with the HMD orientation, the field of view of the HMD 
again shows an entire image captured by the camera. 

Altematively, the camera captures images of areas larger than can be displayed 
and only a portion of the image is displayed. This is considered less preferable since it 
increases the bandwidth requirements and often for no reason as the additional data is not 
30 displayed. 
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Advantageously, when implemented with independent position indicators for each 
of the HMD 1 and the camera 1 1 and independent from the mechanism 13 for moving the 
camera, all types of motion are compensated for including inaccuracies of the mechanism 
13, delays induced by communications, delays induced by the mechanism 13, processing 
5 delays, fine operator motions and so forth. 

When processing is done local to the HMD or on a computer at a same location 
with minima] delays therebetween, each image is accurately aligned on the display within 
a time delay error related only to the processing and the delay in reading HMD sensor 
data. 

10 Thus, discontinuous scene changes are changed into smooth transitions in 

accordance with the expected visual result. 

It is also within the scope of the invention to process the image data prior to 
display thereof in order to determine features or locations within the image data to 
highlight or indicate within the displayed image. For example, contrast may be improved 
15 for generally light or dark images. Also, features may be identified and labeled or 
highlighted. Alternatively, icons or other images are superimposed on the displayed 
image without processing thereof 

Alternatively, the control values are determined in the mechanism for pointing the 
camera instead of by the first computer. In such an embodiment, the HMD position data 
20 is transmitted to the remote system wherein a camera movement related to the HMD 
movement is determined and initiated. 

The above described embodiment compensates for orientation - motion about any 
of three rotational axes. Alternatively, the invention compensates for displacement - 
linear motion along an axis. Further alternatively, the invention compensates for both 
25 linear motion and motion about any of the rotational axes. Displacement and orientation 
are both forms of position and data relating to one or both is referred to here and in the 
claims, which follow, as position data. 

The above described embodiment does not correct images for perspective 
distortion. Doing so is feasible within the concept of time/motion compensation according 
30 to the invention, however it is not generally applicable to use with a single camera, since 
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the depth of field of the observed scene varies. It would require capturing of depth data 
using a range sensor or a three-dimensional vision system. 



Though the above embodiment is described with reference to a physical 
communication link or a wireless communication link between different components, 
5 clearly, either is useful with the invention so long as it is practicable. Also, though the 
HMD is described as a computer peripheral, it could be provided with an internal 
processor and act as a stand alone device. 

According to another embodiment of the invention, areas within the field of view 
that do not correspond to displayed image locations are filled with current image data 

10 relating to earlier captured images for those locations. Preferably, any earlier captured 
images are deemphasized within the field of view in order to prevent the operator from 
being confiised by "stale" image data. For example, each image received fix^m the camera 
is buffered with its associated position data. When some areas within the field of view are 
not occupied by image data, the processor determines another image having image data 

1 5 for those locations within the field of view, the locations determined in accordance with 
the transform performed based on the camera position data associated with the earlier 
captured image and with the current HMD position data. The image data is then displayed 
at the determined location(s) in a ''transparent" fashion. For example, it may be displayed 
with a lower contrast appearing almost ghostlike. Alternatively, the colours are faded to 

20 provide this more ghosdike appearance. Further alternatively, it is displayed identically to 
the current image data. 

The above description is by way of example and is not intended to limit the 
forgoing claims. 
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1 . A method of motion compensation for head mounted displays comprising the steps of: 
providing a captured image from an image capture device to a head mounted display 
5 including a display having a field of view; 

providing camera position data relating to a position of the camera and associated with 
the captured image; 

providing HMD position data relating to a position of the head mounted display; 
transforming the image to vary a displayed location of static objects within the image 
10 relative to the field of view in accordance with the camera position data and the HMD 
position data; and, 

displaying portions of the image at the displayed locations, those portions remaining 
within the field of view. 

15 2. A method according to claim 1 wherein the HMD position data is data relating to the 
present position of the head mounted display and wherein the camera position data is data 
relating to the position of the camera when the associated image is captured and wherein 
the step of transforming the image includes the steps of: 
determining an offset between the HMD position and the camera position; and, 

20 offsetting the image such that it is offset an amount proportional to the offset between the 
HMD position and the camera position. 

3. A method according to claim 2 wherein portions of the field of view for which image 
data is unavailable are filled with a predetermined fill. 

25 

4. A method according to claim 3 wherein the predetermined fill has features for 
maintaining orientation of a wearer of the head mounted display. 

5. A method according to claim 1 wherein the capmred image is larger than the image 
30 necessary to fill the field of view of the head mounted display, and wherein only a portion 

of the captured image is displayed. 

6. A method according to claim 1 wherein the system comprises two head mounted 
displays and two cameras. 

15 
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7. A method according to claim I wherein the system comprises: 

an independent position sensor for sensing the camera position and for providing the 

camera position data. 

5 8. A method according to claim 7 wherein the system comprises: 

an independent position sensor for sensing the head mounted display position and for 
providing the HMD position data, 

9. A method according to claim 1 wherein the system comprises: 

an independent position sensor for sensing the head mounted display position and for 
1 0 providing the HMD position data. 

10. A method according to claim 1 wherein the system comprises: 
a mechanism for moving the camera; and 

means for transmitting the HMD position data to a system in communication with the 
mechanism for moving the camera, 
15 wherein the mechanism for moving the camera moves the camera in response to a change 
in HMD position data, 

1 1 . A method according to claim 1 wherein the HMD position data comprises orientation 
data. 

20 12. A method according to claim 11 wherein the camera position data comprises 
orientation data. 

13. A method according to claim 12 wherein the camera position data comprises 
displacement data. 

25 

14. A method according to claim 1 wherein the HMD position data comprises 
displacement data. 

15. A method according to claim 14 wherein the camera position data comprises 
30 displacement data. 



16 



SUBSTITUTE SHEET (RULE 26) 



wo 01/22149 PCT/CAOO/01063 

16. A method according to claim 15 wherein the camera position data comprises 
orientation data. 



17. A method according to claim 1 comprising the step of transforming the image to 
5 reduce perspective distortion, wherein the camera includes a range sensor. 

18. A motion compensation apparatus for head mounted displays comprising: 

a head mounted display for displaying portions of an image at displayed locations, those 
portions remaining within the field of view; 
10 an image capture device for providing a captured image to a head mounted display 
including a monitor having a field of view; 

a sensor for providing camera position data relating to a position of the camera and 
associated with the captured image; 

a sensor for providing HMD position data relating to a position of the head mounted 
15 display; and 

a processor for transforming the image to vary a displayed location of static objects 
within the image relative to the field of view in accordance with the camera position data 
and the HMD position data. 

20 19. A motion compensation apparatus for head mounted displays according to claim 18 
wherein the image capture device comprises a range sensor. 

20. A motion compensation apparatus for head mounted displays according to claim 19 
wherein the range sensor includes a stereoscopic imaging system. 

25 

21 . A method according to claim 2 comprising the step of: 

when portions of the field of view for which image data is unavailable are detected, 
determining earlier captured image data that when transformed in dependence upon its 
associated camera position data and the HMD position data is for display within the 
30 portions; and, 

displaying with those portions of the field of view the transformed earlier captured image 
data. 
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22. A method according to claim 21 wherein the step of displaying transformed earlier 
captured image data includes a step of de-emphasising the transformed earlier captured 
image data. 
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Fig. 3d 

Rotation of Image 
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